Controlled Experiments on Pair Programming: Making Sense of Heterogeneous Results

Recently, several controlled experiments on adoption and use of pair programming (PP) have been reported. Most of these experiments have been conducted by constructing small-scale programs of a few hundred lines of code, using correctness, duration, and effort metrics. However, none of these experiments is a replication of a previous one; and there are significant differences in contextual factors such as local goals, tasks (“treatments”), subject selection and pairing, defined metrics, and collected measures. Hence, it is very difficult to compare, assess, and generalize the results from such experiments. We illustrate this situation by comparing metrics and measures from two well-known PP experiments. We also discuss a published, formal meta-analysis of 18 PP experiments, including the two compared in this study, and which found PP effects of 10-15 % on key factors like program correctness. We then show how the author of the second PP experiment failed in applying large-scale, commercial defect rates and costs on the small-scale software systems from the author’s PP experiment. We finally argue that: 1) there should be more cooperation and standardization between experimental researchers that investigate the quantitative effects of PP e.g. on program correctness; 2) researchers should to a larger degree apply qualitative methods to study the social and cognitive impact of PP on teamwork, and how professional teams acquire and share knowledge to develop and maintain quality software. Such investigations will require longitudinal case studies in commercial settings, which cannot be achieved through experiments of short duration in academic environments.